The current real-time semantic segmentation algorithm has the high computational cost and large memory footprint， which cannot meet the applications requirements of actual scenes. In order to solve the problems， a new type of shallow lightweight real-time semantic segmentation algorithm — AEFNet （Real-time segmentation algorithm based on Attention mechanism and Effective Factorized convolution） was proposed. Firstly， one-dimensional non-bottleneck structure （Non-bottleneck-1D） was adopted to construct a lightweight factorized convolution module to extract rich contextual information and reduce the amount of calculation. At the same time， the learning ability of the algorithm was enhanced in a simple way and the extraction of detailed information was facilitated. Then， the pooling operation and Attention Refinement Module （ARM） were combined to construct a global context attention module to capture global information and refine each stage of the algorithm to optimize the segmentation effect. The algorithm was verified on the public datasets cityscapes and camvid， and the precision of 74.0% and the inference speed of 118.9 Frames Per Second （FPS） were obtained on the cityscapes test set. Compared with Depth-wise Asymmetric Bottleneck Network （DABNet）， the proposed algorithm has the precision increased by about 4 percentage points， and the inference speed increased by 14.7 FPS. Compared with the recent efficient Enhanced Asymmetric Convolution Network （EACNet）， the proposed algorithm has the precision slightly lower by 0.2 percentage points， but has the inference speed increased by 6.9 FPS. Experimental results show that the proposed algorithm can more accurately identify the scene information， and can meet the real-time requirements.